Multimedia data routing system and method

ABSTRACT

A technique for providing routing of various multimedia events throughout the course of a multimedia presentation using a computer with a storage and a display. A variety of multimedia objects are defined in the storage and grouped in logical relationships to enable multimedia presentations. A display is used to create the presentations interactively by positioning objects representative of the multimedia events and joining them with geometric figures, such as line segments. Each object can then be directly manipulated via a mouse or other pointing device to position a multimedia object to a particular position, or adjust the playback rate of a multimedia object.

This application is a continuation of U.S. patent application Ser. No.09/196,861, filed on Nov. 20, 1998, now U.S Pat. No. 6,421,692, which isa continuation of U.S. patent application Ser. No. 08/120,270, filed onSep. 13, 1993, now abandoned.

COPYRIGHT NOTIFICATION

Portions of this patent application contain materials that are subjectto copyright protection. The copyright owner has no objection to thefacsimile reproduction by anyone of the patent document or the patentdisclosure, as it appears in the Patent and Trademark Office patent fileor records, but otherwise reserves all copyright rights whatsoever.

FIELD OF THE INVENTION

This invention generally relates to improvements in computer systems andmore particularly to a system for routing data between multimediacomponents.

BACKGROUND OF THE INVENTION

Multimedia is perhaps the fastest growing application for computersystems. Increasingly, users are employing computers to present graphic,sound and imaging information to end users. Users are increasinglydemanding ergonomic interfaces for managing multimedia presentations. Inthe past, a time matrix and programming language were used to implementa multimedia presentation. However, simulating a flexible mixing boardto enable the presentation of music or sound with the display ofinformation as a multimedia presentation unfolded was not possible.

Examples of current multimedia systems that do not have the capabilityof the subject invention are Apple's Quicktime and Microsoft's Video forWindows as described in the March issue of NEWMEDIA, “It's Showtime”,pp. 36-42 (1993). The importance of obtaining a solution to the routingproblem encountered in the prior art is discussed in the March issue ofIEEE Spectrum, “Interactive Multimedia”, pp. 22-31 (1993); and “TheTechnology Framework”, IEEE Spectrum, pp. 32-39 (1993). The articlespoint out the importance of an aesthetic interface for controllingmultimedia productions.

SUMMARY OF THE INVENTION

Accordingly, it is a primary objective of the present invention toprovide a system and method for routing multimedia data throughout thecourse of a multimedia presentation using a computer with a storage anda display. A multimedia data routing system is defined in the storageand displayed as a multimedia object. The routing can then be directlymanipulated via a mouse or other pointing device to associate multimediainformation with one or more multimedia objects. A variety of multimediaobjects are defined in the storage and joined in logical relationshipsto enable multimedia presentations. A display is used to create thepresentations interactively by positioning objects representative of themultimedia events and joining them with geometric figures, such as linesegments. Each object can then be directly manipulated via a mouse orother pointing device to position a multimedia object to a particularposition, or adjust the playback rate of a multimedia object.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a personal computer system in accordancewith a preferred embodiment;

FIG. 2 illustrates a prior art, simple, home studio setup utilizing atape deck, mixer, reverberation unit, a pair of microphones, and a pairof speakers;

FIG. 3 is an illustration of a media component in accordance with apreferred embodiment;

FIG. 4 is an illustration of an audio player component in accordancewith a preferred embodiment;

FIG. 5 is an illustration of fan-in and fan out ports in accordance witha preferred embodiment;

FIG. 6 is an illustration of an audio component with zero or more audioinput ports and zero or more audio output ports in accordance with apreferred embodiment;

FIG. 7 is an illustration of a voice annotation application enabled byusing an audio player to record and play voice annotations in accordancewith a preferred embodiment;

FIG. 8 is an illustration of a voice mail/phone answering application inaccordance with a preferred embodiment;

FIG. 9 is an illustration of a multimedia player that is externallysynchronized to a master clock in accordance with a preferredembodiment;

FIG. 10 is an illustration of three sounds, one for music, one for soundeffects, and one for voice over, being mixed together and fed through anoutput device, such as a speaker in accordance with a preferredembodiment;

FIG. 11 is an illustration of some of the audio types supported inaccordance with a preferred embodiment;

FIG. 12 is an illustration of a conversion process, to convert from oneaudio type to another in accordance with a preferred embodiment;

FIG. 13 is an illustration of a remote procedure call in accordance witha preferred embodiment;

FIG. 14 is an illustration of an audio processor architecture with anassociated Run( ) member function which reads data from its inputbuffers, processes it, and writes the result to its output buffers inaccordance with a preferred embodiment;

FIG. 15 is an illustration of audio processors in accordance with apreferred embodiment;

FIG. 16 illustrates how processing is performed for the network shown inFIG. 15 in accordance with a preferred embodiment;

FIG. 17 illustrates an audio port in accordance with a preferredembodiment;

FIG. 18 is an illustration of an audio processor, such as a player, inaccordance with a preferred embodiment;

FIG. 19 is a flowchart of the recursive logic associated with activatingan audio processor in accordance with a preferred embodiment;

FIG. 20 is a flowchart setting forth the detailed logic associated withdeactivating the audio processor in accordance with a preferredembodiment;

FIG. 21 is a flowchart setting forth the logic associated with runningan audio processor in accordance with a preferred embodiment;

FIG. 22 is an example of patching a video digitizer component to aviewer component for display on a computer's screen in accordance with apreferred embodiment;

FIG. 23 is an example of mixing images from two video objects in aneffects processor and displaying the result on a computer screen inaccordance with a preferred embodiment;

FIG. 24 illustrates how graphic ports are used in accordance with apreferred embodiment;

FIG. 25 is a flowchart presenting the logic associated with an outputport's Write member function in accordance with a preferred embodiment;

FIG. 26 is an illustration of an input port's read processing inaccordance with a preferred embodiment;

FIG. 27 illustrates an input port's next member processing in accordancewith a preferred embodiment;

FIG. 28 shows how two components, a MIDI player 2800 and a MIDIinterface 2810, can be used to play a music synthesizer connected to thecomputer in accordance with a preferred embodiment;

FIG. 29 shows how MIDI data can be recorded and played back from anexternal music synthesizer in accordance with a preferred embodiment;

FIG. 30 illustrates how MIDI data can be played back in accordance witha preferred embodiment;

FIG. 31 shows a media component that contains both MIDI and audio portsin accordance with a preferred embodiment;

FIG. 32 illustrates the detailed system messages divided into common,real-time, and exclusive messages in accordance with a preferredembodiment;

FIG. 33 is an illustration of some of the formats of MIDI messages inaccordance with a preferred embodiment;

FIG. 34 is an illustration of a MIDI packet object encapsulating MIDImessage types and structures, a status byte and a time stamp inaccordance with a preferred embodiment;

FIG. 35 illustrates an example of a fanning operation in accordance witha preferred embodiment;

FIG. 36 is an illustration of a MIDI output port's Write( ) memberfunction in accordance with a preferred embodiment;

FIG. 37 is an illustration of a MIDI input port's Read( ) memberfunction in accordance with a preferred embodiment;

FIG. 38 illustrates a media component having a single input port and twooutput ports in accordance with a preferred embodiment;

FIG. 39 presents an example of a media component that can playtime-based media sequences in accordance with a preferred embodiment;

FIG. 40 illustrates an audio player for an audio component to play andrecord audio data in accordance with a preferred embodiment;

FIG. 41 illustrates an example of a speaker component in accordance witha preferred embodiment;

FIG. 42 illustrates a microphone component in accordance with apreferred embodiment;

FIG. 43 illustrates an example of a mixer component in accordance with apreferred embodiment;

FIG. 44 illustrates an example of a splitter component in accordancewith a preferred embodiment;

FIG. 45 illustrates an example of a gain component in accordance with apreferred embodiment;

FIG. 46 illustrates an echo component in accordance with a preferredembodiment;

FIG. 47 illustrates a fuzz component in accordance with a preferredembodiment;

FIG. 48 illustrates an example of an audio type converter in accordancewith a preferred embodiment;

FIG. 49 illustrates an audio multiconverter in accordance with apreferred embodiment;

FIG. 50 illustrates a sound component in accordance with a preferredembodiment;

FIG. 51 illustrates the components imbedded in a sound component with apreferred embodiment;

FIG. 52 illustrates a physical speaker component in accordance with apreferred embodiment;

FIG. 53 illustrates a physical microphone component in accordance with apreferred embodiment;

FIG. 54 illustrates a graphic player component in accordance with apreferred embodiment;

FIG. 55 illustrates a graphic viewer component in accordance with apreferred embodiment;

FIG. 56 illustrates a video digitizer component in accordance with apreferred embodiment;

FIG. 57 illustrates a MIDI player component in accordance with apreferred embodiment;

FIG. 58 illustrates a MIDI interface component in accordance with apreferred embodiment;

FIG. 59 illustrates a MIDI flter component in accordance with apreferred embodiment;

FIG. 60 illustrates a MIDI mapper component in accordance with apreferred embodiment;

FIG. 61 illustrates a MIDI program mapper component in accordance with apreferred embodiment;

FIG. 62 illustrates a MIDI note mapper component in accordance with apreferred embodiment; and

FIG. 63 illustrates a MIDI channel mapper component in accordance with apreferred embodiment.

FIG. 64 illustrates a class diagram that shows the relationships betweenthe multimedia object, multimedia component and connecting objectclasses and related subclasses.

DETAILED DESCRIPTION OF THE INVENTION

The invention is preferably practiced in the context of an operatingsystem resident on a personal computer such as the IBM® PS/2® or Apple®Macintosh® computer. A representative hardware environment is depictedin FIG. 1, which illustrates a typical hardware configuration of aworkstation in accordance with the subject invention having a centralprocessing unit 10, such as a conventional microprocessor, and a numberof other units interconnected via a system bus 12. The workstation shownin FIG. 1 includes a Random Access Memory (RAM) 14, Read Only Memory(ROM) 16, an I/O adapter 18 for connecting peripheral devices such asdisk units 20 to the bus, a user interface adapter 22 for connecting akeyboard 24, a mouse 26, a speaker 28, a microphone 32, and/or otheruser interface devices such as a touch screen device (not shown) to thebus, a communication adapter 34 for connecting the workstation to a dataprocessing network and a display adapter 36 for connecting the bus to adisplay device 38. The workstation has resident thereon an operatingsystem such as the Apple System/7® operating system.

In a preferred embodiment, the invention is implemented in the C++programming language using object oriented programming techniques. Aswill be understood by those skilled in the art, Object-OrientedProgramming (OOP) objects are software entities comprising datastructures and operations on the data. Together, these elements enableobjects to model virtually any real-world entity in terms of itscharacteristics, represented by its data elements, and its behavior,represented by its data manipulation functions. In this way, objects canmodel concrete things like people and computers, and they can modelabstract concepts like numbers or geometrical concepts. The benefits ofobject technology arise out of three basic principles: encapsulation,polymorphism and inheritance.

Objects hide, or encapsulate, the internal structure of their data andthe algorithms by which their functions work. Instead of exposing theseimplementation details, objects present interfaces that represent theirabstractions cleanly with no extraneous information. Polymorphism takesencapsulation a step further. The idea is many shapes, one interface. Asoftware component can make a request of another component withoutknowing exactly what that component is. The component that receives therequest interprets it and determines, according to its variables anddata, how to execute the request. The third principle is inheritance,which allows developers to reuse pre-existing design and code. Thiscapability allows developers to avoid creating software from scratch.Rather, through inheritance, developers derive subclasses that inheritbehaviors, which the developer then customizes to meet their particularneeds.

A prior art approach is to layer objects and class libraries in aprocedural environment. Many application frameworks on the market takethis design approach. In this design, there are one or more objectlayers on top of a monolithic operating system. While this approachutilizes all the principles of encapsulation, polymorphism, andinheritance in the object layer, and is a substantial improvement overprocedural programming techniques, there are limitations to thisapproach. These difficulties arise from the fact that while it is easyfor a developer to reuse their own objects, it is difficult to useobjects from other systems and the developer still needs to reach intothe lower, non-object layers with procedural Operating System (OS)calls.

Another aspect of object oriented programming is a framework approach toapplication development. One of the most rational definitions offrameworks came from Ralph E. Johnson of the University of Illinois andVincent F. Russo of Purdue. In their 1991 paper, Reusing Object-OrientedDesigns, University of Illinois tech report UIUCDCS91-1696 they offerthe following definition: “An abstract class is a design of a set ofobjects that collaborate to carry out a set of responsibilities. Thus, aframework is a set of object classes that collaborate to execute definedsets of computing responsibilities.” From a programming standpoint,frameworks are essentially groups of interconnected object classes thatprovide a pre-fabricated structure of a working application. Forexample, a user interface framework might provide the support and“default” behavior of drawing windows, scrollbars, menus, etc. Sinceframeworks are based on object technology, this behavior can beinherited and overridden to allow developers to extend the framework andcreate customized solutions in a particular area of expertise. This is amajor advantage over traditional programming since the programmer is notchanging the original code, but rather extending the software. Inaddition, developers are not blindly working through layers of codebecause the framework provides architectural guidance and modeling butat the same time frees them to then supply the specific actions uniqueto the problem domain.

From a business perspective, frameworks can be viewed as a way toencapsulate or embody expertise in a particular knowledge area.Corporate development organizations, Independent Software Vendors (ISV)sand systems integrators have acquired expertise in particular areas,such as manufacturing, accounting, or currency transactions. Thisexpertise is embodied in their code. Frameworks allow organizations tocapture and package the common characteristics of that expertise byembodying it in the organization's code. First, this allows developersto create or extend an application that utilizes the expertise, thus theproblem gets solved once and the business rules and design are enforcedand used consistently. Also, frameworks and the embodied expertisebehind the frameworks, have a strategic asset implication for thoseorganizations who have acquired expertise in vertical markets such asmanufacturing, accounting, or biotechnology, and provide a distributionmechanism for packaging, reselling, and deploying their expertise, andfurthering the progress and dissemination of technology.

Historically, frameworks have only recently emerged as a mainstreamconcept on personal computing platforms. This migration has beenassisted by the availability of object-oriented languages, such as C++.Traditionally, C++ was found mostly on UNIX systems and researcher'sworkstations, rather than on computers in commercial settings. It islanguages such as C++ and other object-oriented languages, such asSmalltalk and others, that enabled a number of university and researchprojects to produce the precursors to today's commercial frameworks andclass libraries. Some examples of these are InterViews from StanfordUniversity, the Andrew toolkit from Carnegie-Mellon University andUniversity of Zurich's ET++ framework.

Types of frameworks range from application frameworks that assist indeveloping the user interface, to lower level frameworks that providebasic system software services such as communications, printing, filesystems support, graphics, etc. Commercial examples of applicationframeworks are MacApp (Apple), Bedrock (Symantec), OWL (Borland),NeXTStep App Kit (NeXT), and Smalltalk-80 MVC (ParcPlace).

Programming with frameworks requires a new way of thinking fordevelopers accustomed to other kinds of systems. In fact, it is not like“programming” at all in the traditional sense. In old-style operatingsystems such as DOS or UNIX, the developer's own program provides all ofthe structure. The operating system provides services through systemcalls-the developer's program makes the calls when it needs the serviceand control returns when the service has been provided. The programstructure is based on the flow-of-control, which is embodied in the codethe developer writes.

When frameworks are used, this is reversed. The developer is no longerresponsible for the flow-of-control. The developer must forego thetendency to understand programming tasks in term of flow of execution.Rather, the thinking must be in terms of the responsibilities of theobjects, which must rely on the framework to determine when the tasksshould execute. Routines written by the developer are activated by codethe developer did not write and that the developer never even sees. Thisflip-flop in control flow can be a significant psychological barrier fordevelopers experienced only in procedural programming. Once this isunderstood, however, framework programming requires much less work thanother types of programming.

In the same way that an application framework provides the developerwith prefab functionality, system frameworks, such as those included ina preferred embodiment, leverage the same concept by providing systemlevel services, which developers, such as system programmers, use tosubclass/override to create customized solutions. For example, considera multimedia framework which could provide the foundation for supportingnew and diverse devices such as audio, video, MIDI, animation, etc. Thedeveloper that needed to support a new kind of device would have towrite a device driver. To do this with a framework, the developer onlyneeds to supply the characteristics and behaviors that are specific tothat new device.

The developer in this case supplies an implementation for certain memberfunctions that will be called by the multimedia framework. An immediatebenefit to the developer is that the generic code needed for eachcategory of device is already provided by the multimedia framework. Thismeans less code for the device driver developer to write, test, anddebug. Another example of using system frameworks would be to haveseparate I/O frameworks for SCSI devices, NuBus cards, and graphicsdevices. Because there is inherited functionality, each frameworkprovides support for common functionality found in its device category.Other developers could then depend on these consistent interfaces forimplementing other kinds of devices.

A preferred embodiment takes the concept of frameworks and applies itthroughout the entire system. For the commercial or corporate developer,systems integrator, or OEM, this means all the advantages that have beenillustrated for a framework such as MacApp can be leveraged not only atthe application level for such things as text and user interfaces, butalso at the system level, for services such as graphics, multimedia,file systems, I/O, testing, etc.

Application creation in the architecture of a preferred embodiment willessentially be like writing domain-specific pieces that adhere to theframework protocol. In this manner, the whole concept of programmingchanges. Instead of writing line after line of code that calls multipleAPI hierarchies, software will be developed by deriving classes from thepreexisting frameworks within this environment, and then adding newbehavior and/or overriding inherited behavior as desired.

Thus, the developer's application becomes the collection of code that iswritten and shared with all the other framework applications. This is apowerful concept because developers will be able to build on eachother's work. This also provides the developer the flexibility tocustomize as much or as little as needed. Some frameworks will be usedjust as they are. In some cases, the amount of customization will beminimal, so the piece the developer plugs in will be small. In othercases, the developer may make very extensive modifications and createsomething completely new.

In a preferred embodiment, as shown in FIG. 1, a multimedia data routingsystem manages the movement of multimedia information through thecomputer system, while multiple media components resident in the RAM 14,and under the control of the CPU 10, or externally attached via the bus12 or communication adapter 34, are responsible for presentingmultimedia information. No central player is necessary to coordinate ormanage the overall processing of the system. This architecture providesflexibility and provides for increased extensibility as new media typesare added. The system makes use of a variety of multimedia objects, someof which that can be used as connecting objects. The connecting objectsinclude gain, filters, amplifiers, mixers, players and other multimediacomponents that are individually implemented as objects in an objectoriented operating system. Objects, as discussed above include a code ormethod component and a data component. The system includes a mouse forfacilitating iconic operations such as drag/drop, double-clicking,drop-launching, cursor positioning and other typical operations.

In video and audio production studios, media such as sound, MIDI, andvideo make use of physical patch cords to route signals between sources,effects processors, and sinks. Signal processing algorithms are alsooften represented as networks of sources, sinks, and processors. Both ofthese models can be represented as directed graphs of objects that areconnected. A preferred embodiment allows this model—connecting objectstogether—to be realized on a computer system. FIG. 2 illustrates a priorart, simple, home studio setup utilizing a tape deck, mixer,reverberation unit, pair of microphones, and pair of speakers. Since themicrophones are connected to the tape deck, sound input is routed fromthe microphone to the tape deck, where it can be recorded. When the tapedeck plays back, its signal is routed to the mixer because of theconnection from the tape deck to the mixer. Similarly, the reverberationunit and the speakers are connected an amplifier connected to the mixer.

A preferred embodiment utilizes object-oriented technology to representa connection model. Multimedia objects can be connected to each other,creating directed data flow graphs. In addition, a standard set ofmultimedia objects is defined to the system. These objects can beconnected together to facilitate multimedia data flow from object toobject. The connection operations can be facilitated by connectingmultimedia objects via a geometric figure such as a line, line segmentor other appropriate geometry. The figures discussed below show examplesof various multimedia objects, including connecting objects and thegeometric figures that are used to represent the internal datastructures and logic joining the multimedia objects.

Classes for Routing

A time-based media component (hereafter referred to as a mediacomponent) base class is a central abstraction used for routing. A mediacomponent has zero or more input ports and zero or more output ports. InFIG. 3, for example, the media component has a single input port 300 andtwo output ports 310 and 320. Ports 300, 310 and 320 are represented asfilled triangles.

Subclasses of media components are connected together by connectingtheir ports. This processing is analogous to using patch cords toconnect audio and video components together in a recording studio. InFIG. 4, a subclass of a media component, an audio player componentobject, is connected to another media component subclass, a speakercomponent object. The audio player has one output port and the speakerhas one input port. Media components are controlled using memberfunction calls. The audio player in FIG. 4, for example, has memberfunctions for playing audio data. When the audio player's memberfunction Play( ) is called, audio data will be fed from the audio playerto the speaker, which will cause the audio to be heard on the computer'sspeaker. The speaker component does not have a Play( ) function becauseit plays whatever audio data is transmitted to it. Media components canbe implemented completely in software. However, it is possible for amedia component to represent a physical piece of hardware. For example,a speaker object can be employed to represent playback hardware of acomputer. In this way, external media devices, such as video taperecorders, mixers, and effect processors, can be represented as mediacomponents and be connected together.

Connecting Media Components

Media components are connected together by connecting their ports. Toprevent client objects and components from writing data simultaneouslyto the same port, thereby compromising data integrity, clients are notallowed to access ports directly. Instead, clients perform connectionoperations on multithread-safe surrogate objects. In the context of thisdescription, the term “surrogate” refers to a specialized representationof an underlying class which permits multiple clients to share instancesof the class safely. In the case of the media components, surrogateports permit limited indirect manipulation of the actual ports. Everymedia component has member functions which create surrogates for each ofits input and output ports. These port surrogates are extremelylightweight objects and are well-suited for traversing addressboundaries, thus facilitating the connection of media components whichreside in different address spaces.

Each port and port surrogate has a data type associated with it.Examples of types are MIDI data, 44 kHz 16 bit audio data, 22 kHz 8 bitaudio data, and graphic data for video. When two ports are asked toconnect, a type negotiation protocol insures that the ports are capableof supporting compatible data types. An exception is generated if theports have no types in common.

Converter objects can be inserted between objects which haveincompatible port types. Converters are components which take in data ofone type in order to produce data of a different type. Examples offan-in and fan-out ports are shown in FIG. 5. Specific subclasses maychoose to disallow fan-in and/or fan-out. Audio ports, for example,disallow both, and exceptions are generated by any attempt to connectone port to more than one other port. Fan-in and fan-out properties arehandled by the specific media subclasses. For example, in MIDI, fan-outsends the same data to multiple recipients, and fan-in merges data frommultiple senders.

Two media components, A and B, are connected together by:

-   1) calling a member function of A to request an output port    surrogate, Pa;-   2) calling a member function of B to ask for B's to request an input    port surrogate, Pb;-   3) calling Pa's member function, ConnectTo, and passing in Pb as an    argument.    The ports can be disconnected by calling Pa's member function,    DisconnectFrom, and passing in Pb as an argument. The above    procedures are invoked when connecting or disconnecting any media    components, whether the port type is audio, graphic, MIDI or some    other multimedia data type. There is no compile-time checking when    two ports are connected. For example, there is nothing to prevent a    software developer from writing, compiling, and linking code which    attempts to connect an audio port with a MIDI port. Instead, an    exception is thrown at run time. This behavior has been explicitly    designed into the routing system in order to enable polymorphic    connection of media ports. A patching module, for example, could    connect pairs of media ports together at specific connect times    without having to give special treatment to different port types.    Audio, video, and MIDI ports would all be handled in exactly the    same way by a patcher. Connection of objects can be represented    visually by drawing a line segment between indicia representative of    the multimedia objects such as an icon on the display.

Routing Audio Data

Audio can be digitized, stored, processed, and played back on acomputer. A computer can also be used to synthesize and playback audio.An Analog to Digital Converter (ADC) is used to convert an analog audioelectrical signal to a series of numbers called digital audio samples.The audio is typically sampled at rates of 8000 samples per second fortelephone quality audio all the way up to 44,100 samples per second forCompact Disc (CD) quality audio and 48,000 samples per second forDigital Audio Tape (DAT). Once in numeric form, the audio can be storedand processed by the computer. The digital audio samples are convertedback to an analog audio electrical signal by using a Digital to AnalogConverter (DAC). A preferred embodiment defines subclasses of portobjects called audio ports that are used to route digital audio databetween media components. Audio output ports can be connected to audioinput ports to route audio data. If a media component has at least oneaudio port, it is called an audio component. All audio components aresubclasses of media component base class. An audio component has zero ormore audio input ports and zero or more audio output ports, as shown inFIG. 6. Audio components can be connected together by connecting theirports. This is analogous to using patch cords to connect stereocomponents together utilizing hardware.

A preferred embodiment facilitates the connection of audio components tocreate a variety of interesting applications. FIGS. 7, 8, 9 and 10illustrate some example applications that can be constructed using audiocomponents. FIG. 7 shows how a voice annotation application can bewritten by using an audio player to record and play voice annotations. Atelephone handset is used for input and output. FIG. 8 shows how a voicemail/phone answering application can be constructed. One audio playerplays a greeting out over the phone line while another audio playerrecords an incoming message. FIG. 9 illustrates a music application.Echo, a special effect, is added to a musical instrument sound andplayed through a speaker. FIG. 10 illustrates how three sounds, one formusic, one for sound effects, and one for voice over, can be mixedtogether and fed through an output device, such as a speaker.

Like all media components, audio components are connected together byconnecting their ports. This operation is facilitated by selecting anaudio component port, extending a geometric figure such as a line fromthe component port to another multimedia component port and creating adata structure commemorating the linkage represented graphically on thedisplay. Every audio component has member functions which createsurrogates for its input and output ports. Clients perform connectionoperations by requesting input and/or output port surrogates from eachcomponent and then using the member functions provided by the surrogateobjects to connect the actual input ports to actual output ports. Eachaudio port and port surrogate has an audio type associated with it. FIG.11 lists some of the audio types supported by a preferred embodiment.

A connection, represented by the linking line segment on the display,represents a single audio channel. Stereo is handled by using twoconnections, one for the left channel and one for the right. When twoports are asked to connect, a type negotiation protocol insures that theports are capable of supporting compatible data types. An exception isgenerated if the ports have no types in common. If this happens, anaudio type converter can be inserted between the two ports to convertfrom one type to another as illustrated in FIG. 12. No loops (cycles)are allowed in a network of audio components. Any attempt to perform aconnection that would result in a loop causes an exception to be thrown.The process of connecting audio components is represented graphically onthe display by extending a geometric figure, such as a line segment,between icon like figures representative of the audio input and outputports of indicia representative of multimedia objects.

Implementing Audio Components

A client-server based architecture is used to implement audiocomponents. For every audio component, there is an audio processorobject that resides in a task called the sound server. The audioprocessor object is responsible for performing the actual signalprocessing. Audio component subclassers have to write both an audiocomponent and an audio processor subclass. A client/server approach isused to enhance performance. Patch cord-style routing of audio is mostefficiently implemented if all data movement and signal processing aredone in a single task. This avoids unnecessary InterprocessCommunication (IPC) and context switches. Given that clients exist inmany different tasks, a sound server is required to centralize theprocessing in a single task.

An alternative implementation has been explored which handles signalprocessing in a separate task per audio component. No server is neededfor audio processing. This is an elegant alternative with manyadvantages, but unfortunately it has one drawback—it is over an order ofmagnitude slower than the approach presented in a preferred embodimentof the invention. On monoprocessors, this ratio would remain fairlyconstant even as machines are built with enhanced processing power.

Client/Server Model

Audio components require facilities for communicating with theircorresponding audio processors. A Remote Procedure Call (RPC) interfaceis used to do this. Generally, member functions of an audio componentsubclass remotely invoke the “real” member function on the audioprocessor. FIG. 13 is a block diagram illustrating a remote procedurecall in accordance with a preferred embodiment.

Making Connections

Audio processors, in contrast to audio components, own the audio ports.Audio components own the audio port surrogates. Whenever two portsurrogates are connected together in the client address space, thecorresponding audio ports are connected together in the sound serveraddress space. Subclassers are not required to take any enabling action,the framework does it for them.

Processing Audio

Connections between audio ports are implemented using buffers of audiodata. Each connection, or “patch cord,” has a buffer associated with it.The audio processor can ask each of its audio ports for the address andsize of this buffer. FIG. 14 is a block diagram illustrating an audioprocessor architecture with an associated Run( ) member function whichreads data from its input buffers, processes it, and writes the resultto its output buffers. The size of the buffers is variable but thepreferred size accommodates 5 milliseconds worth of samples. This sizeis necessary to reach a performance goal of being able to start and stopsounds with a maximum of 10 ms latency.

Frame-Based Processing

The sound server uses a technique called frame-based processing toprocess sound samples. The basic process is presented below.

-   1) Order the audio processors so that producers come before    consumers.-   2) For each audio processor, call it's Run( ) member function.    Repeat this step whenever an I/O device requires data.    This is an extremely efficient way to process audio, because once    the running order is figured out, frames can be produced cheaply by    calling Run( ). Run( )'s implementation can be very efficient,    because it utilizes fixed size buffers that can be completely    processed in a single frame. For example, consider FIG. 15, which    illustrates a network of audio processors. Each audio processor, or    node, is labeled with a letter.    To produce sound, the sound server performs the following steps:-   1) Orders the audio processors so that producers come before    consumers; and-   2) Runs them whenever an audio I/O device, such as the speaker,    needs more data.    FIG. 16 illustrates how this processing is performed for the network    shown in FIG. 15. In the first block the audio processors are    ordered, then the processors are run in the specified order. In    parallel, information is obtained for each of the processors. No    processor is run before its time, in other words, until the    appropriate information is available for the processor, the    processor enters a wait state.

Ordering Audio Processors

Audio processors are ordered employing a technique called simulated dataflow. Most subclassers don't need to worry about ordering becauseordering is determined automatically by a framework in most cases.Simulated data flow iterates through the network in topological order,as if to pass data from one processor to the next. This is done to seewhich audio processor can run and which audio processor can't. Audioprocessors that run are put on the run list, and are utilized over andover again during the run phase.

An audio processor node can run if:

-   #1: its input ports has data available, AND-   #2: its output ports have a place to put data.    Each audio processor has a CanRun( ) member function, which returns    TRUE if the node can run. CanRun( )'s default implementation uses    the above rules, so a subclasser only has to override CanRun( ) if    the audio processor needs to override a rule. If an audio processor    can run, it simulates the flow of data out on all output ports.    Fire( )actually just turns around and calls another Fire( ) member    function on each output port. Firing an output port simulates the    flow of data out the port, making simulated data available on the    input port on the other side. Input ports have an IsAvailable( )    member function, which returns TRUE if the port has simulated data    available. FIG. 17 illustrates an audio port in accordance with a    preferred embodiment.

Delays must take priority over CanRun( ) because the input to a delaygoes silent. Therefore, the delay must still produce output until thedelay is exhausted. The input port no longer has data available but theaudio processor must still run. CanRun( ) must be modified to returnTRUE when no input is available and there is still data in the delaythat must be output. Players must also override CanRun( ), because whena player is stopped, a player can't run, even if the dataflow criteriaare met. Players must modify the rules so that CanRun( ) always returnsFALSE if the player is stopped, regardless of whether dataflow criteriaare met or not. An audio processor can query its audio output port toascertain the delay from when the data is written to the port's bufferto when the audio will actually be heard by the user. This allows audioprocessors to synchronize to arbitrary time bases. FIGS. 18, 19 and 20are flowcharts illustrating the logic associated with audio processors.An audio processor, such as a player, determines that execution isrequired and invokes RequestOrder, which is depicted in FIG. 18.

FIG. 18 is a flowchart presenting the detailed logic for calling anorder request in accordance with a preferred embodiment. Processingcommences at terminal 1800 and immediately passes to function block 1810to mark the run list invalid. The run list can be marked invalid by droplaunching a multimedia object, double-clicking on a multimedia object orusing any other iconic operation to signify initialization of anoperation. Then, at decision block 1820, a test is performed todetermine if the audio processor can run. If so, then at function block1830, the audio processor is deactivated and control is passed toterminal 1850. However, if the audio processor cannot run, then infunction block 1840, the audio processor is activated and control passesto terminal 1850.

FIG. 19 is a flowchart of the recursive logic associated with activatingan audio processor in accordance with a preferred embodiment. Processingcommences at terminal 1900 where control is passed from function block1840 or a recursive call is made from terminal 1990. In either case,control is immediately passed to function block 1910 where index i isequated to zero and counter N is set equal to the number of audioprocessor output ports. Then, a test is performed at decision block 1920to determine if counter i=N. If so, then control is returned at terminal1930. If not, then output port i is fired at function block 1940, theoutput port is marked as having fired at function block 1950, the audioprocessor connected to the output port is obtained at function block1960, counter i is incremented at function block 1970, and a test isperformed at decision block 1980 to determine if the audio processor canrun. If the audio processor can run, then a recursive call to the logicin FIG. 19 is placed via terminal 1900. If not, then control is passedto decision block 1920 to continue processing.

FIG. 20 is a flowchart setting forth the detailed logic associated withdeactivating the audio processor. Processing commences at terminal 2000where control is passed from function block 1830, or a recursive call ismade from terminal 2018 or 2080. In either case, control is immediatelypassed to function block 2004 where index i is equated to zero andcounter N is set equal to the number of audio processor output ports.Then, a test is performed at decision block 1920 to determine if counteri=N. If so, then control is passed to function block 2020 toreinitialize the index and counter. Then, at decision block 2024, a testis performed to determine if i=N. If not, then input port i is marked ashaving not fired at function block 2040, the audio processor connectedto the input port is obtained at function block 2050, counter isincremented at function block 2060, and a test is performed at decisionblock 2070 to determine if the audio processor can run. If so, thencontrol is returned at terminal 2024. If the audio processor cannot run,then a recursive call to the logic in FIG. 20 is placed via terminal2080. If the audio processor can run, then control is passed to decisionblock 2024 to test the index again.

If index i does not equal to N at decision block 2006, then the outputport i is marked as having NOT fired in function block 2008, the audioprocessor connected to the output port is obtained at function block2010, counter i is incremented at function block 2012, and a test isperformed at decision block 2014 to determine if the audio processor canrun. If the audio processor cannot run, then a recursive call to thelogic in FIG. 20 is placed via terminal 2018. If not, then control ispassed to decision block 2006 to test the index again and continue theprocessing.

FIG. 21 is a flowchart setting forth the logic associated with runningan audio processor in accordance with a preferred embodiment. Processingcommences at terminal 2100 and immediately passes to decision block 2104to determine if the run list is valid. If the run list is not valid,then a topological sort is performed in function block 2106 to sort theaudio processors in fired order. Then, in function block 2110, thesorted list is marked as valid, and control passes to function block2120. If the run list was not valid at decision block 2104, then controlpasses to function block 2120 to reset the index i and initialize thecount N. Next, a test is performed at decision block 2122 to determineif the counter i has reached the count N. If so, then processing iscompleted and control passes to terminal 2130. If not, then audioprocessor i is run as depicted in function block 2124, and the index iis incremented as depicted in function block 2126.

Routing Video and Graphical Data

Video information can be digitized, stored, processed, and played backon a computer. A video digitizer is used to convert an analog videoelectrical signal to a series of digital images, called a frame. Thenumber of frames digitized per second is called the frame rate. Fifteento thirty frames a second are typical frame rates. Once in digital form,the video can be stored and processed by the computer. The video can beplayed back by displaying the digital images in sequence at the originalframe rate on a computer screen.

A graphic object is a base class used to represent any object that canbe displayed on a computer screen. Subclasses of the graphic object,include, but are not limited to polygons, curves, and digital images.Each frame of digitized video is a digital image, and hence can berepresented by a graphic object.

Graphic input and output ports are used to route graphic objects fromone media component to another. Digital video can be routed this way, aseach video frame is a graphic object. Animation data can also be routedwith graphic ports, because graphic ports can route any graphic object.Using media components containing graphic ports, it is possible tocreate networks of video objects with video streaming between them. Avariety of interesting applications can be constructed this way. FIG. 22illustrates an example of patching a video digitizer component to aviewer component for display on the computer's screen in accordance witha preferred embodiment.

FIG. 23 is a more complicated example of mixing images from two videoobjects in an effects processor and displaying the result on a computerscreen in accordance with a preferred embodiment. Graphic ports areconnected together by means of port surrogates. Every video componenthas member functions which create surrogates for its input and outputports. Clients perform connection operations by requesting input and/oroutput port surrogates from each component. Then, using the memberfunctions provided by the surrogate objects connect the actual inputports to actual output ports. Each graphic port and port surrogate has agraphic type associated with it. When two ports are asked to connect, atype negotiation protocol insures that the ports are capable ofsupporting compatible data types. An exception is generated if the portshave no types in common.

The process of connecting video components can be representedgraphically on a display by extending a geometric figure such as a linesegment between the indicia representative of a video input and outputports.

Routing Data

Graphic ports manipulate graphic data by reading and writing pointers tographic objects. The port implementation for writing is synchronous: thegraphic output port's Write( ) member function blocks until the receiveris done with the graphic object being sent. The graphic port does notemploy copy semantics for performance reasons. Copying RGB bit-mappedimages is avoided because of the amount of processor and storageinvolved in such operations. Instead, a pointer to the graphic object issent from the graphic output port to the graphic input port. Thesynchronous interface functions across address spaces if shared memoryis used for storing the graphic objects being written and read, or ifthe graphic objects are copied across a task's address space. Blockedgraphic ports are unblocked in the event that a connection is broken.

FIG. 24 illustrates how graphic ports are used in accordance with apreferred embodiment. 1) A task in the source media component writes agraphic object to the media component's output port. 2) A pointer to thegraphic object is transferred to the connected input port of thedestination media component. 3) A task in the destination mediacomponent, having called it's input port's Read( ) member function andblocked, unblocked and read the graphic object pointer. 4) When thistask is finished processing the graphic object, it calls the Next( )member function of the destination media component's input port. 5) Thesource media component's task unblocks, returning from the write call.It can now safely dispose of the graphic object because the destinationis finished with it. FIG. 25 is a flowchart presenting the logicassociated with an output port's Write member function. A pointer to agraphic object is passed into this member function. Processing commencesat terminal 2500 and immediately passes to decision block 2510 todetermine if the port is connected. If the port is not connected, thenan exception occurs as shown at terminal 2520. If a port is connected,then a test is performed at decision block 2530 to determine if theconnected port is in the same address space. If the port is not in thesame address space, then at function block 2540, the entire graphicobject is copied into shared memory. If the port is in the same addressspace, then a pointer to the graphic object is copied into memory. Ineither case, the next step is to send a notification to the input portas depicted at function block 2560, block the task until the input portnotifies the port processing is completed as depicted at function block2570, and terminating at terminal 2580.

FIG. 26 illustrates an input port's read processing in accordance with apreferred embodiment. Processing commences at terminal 2600 andimmediately passes to decision block 2610 to determine if the port isconnected. If the port is not connected, then the framework generates anexception at terminal 2620. If the port is connected, then another testis performed at decision block 2630 to determine if the graphic isready. If not, then a block task is processed until the object is readyas depicted in function block 2640, and control is passed to functionblock 2650. If the graphic is ready, then a pointer is returned to thegraphic object in function block 2650 and control is returned viaterminal 2660.

FIG. 27 illustrates an input port's next member processing in accordancewith a preferred embodiment. Processing commences at terminal 2700 andimmediately passes decision block 2710 to determine if the port isconnected. If the port is not connected, then an exception occurs asshown in terminal 2720. If the port is connected, then an appropriatenotification is transmitted as shown in function block 2730, and anothertest is performed at decision block 2740 to determine if the connectedport is in the same address space. If the port is in the same addressspace, then processing is completed at terminal 2760. If the port is notin the same address space, then a copy of the graphic object is deletedas shown in function block 2750 and processing is terminated at terminal2760.

Routing MIDI Data

Musical Instrument Digital Interface (MIDI) defines an interface forexchanging information between electronic musical instruments,computers, sequencers, lighting controllers, mixers, and tape recordersas discussed in MIDI Manufacturers Association publication entitled,MIDI 1.0 Detailed Specification (1990). MIDI is extensively used both inthe recording studio and in live performances and has had enormousimpact in the areas of studio recording and automated control, audiovideo production, and composition. By itself and in conjunction withother media, MIDI plays an integral role in the application of computersto multimedia applications. In comparison to digital audio, MIDI filestake up much less space, and the information is symbolic for convenientmanipulation and viewing. For example, a typical 3 minute MIDI file mayrequire 30 to 60 Kilobytes on disk, whereas a CD-quality, stereo audiofile requires about 200 Kilobytes per second, or 36 Megabytes for 3minutes. MIDI data may appear as musical notation, graphical piano-roll,or lists of messages suitable for editing and reassignment to differentinstruments. General MIDI has standardized instrument assignments togreatly motivate the multimedia title producer.

MIDI input and output ports are used to route time-stamped MIDI packetsfrom one media component to another. MIDI ports act as mailboxes for thecommunication of MIDI packets across address spaces. Many interestingMIDI applications can be created by connecting media components thatcontain MIDI ports. FIG. 28 illustrates how two components, a MIDIplayer 2800 and a MIDI interface 2810, can be used to play a musicsynthesizer connected to the computer. The MIDI interface is used toconnect external devices, such as a music synthesizer. MIDI packets aresent from the MIDI player to the MIDI interface. The MIDI interface 2810converts the MIDI packets to MIDI data which is sent to the musicsynthesizer for playback.

FIG. 29 shows how MIDI data can be recorded and played back from anexternal music synthesizer. The MIDI interface 2910 has a MIDI outputport that produces MIDI packets based upon data received from the musicsynthesizer. The MIDI player 2900 has a MIDI input port that reads thesepackets and stores them on the computer. FIG. 30 illustrates how MIDIdata can be played back 3000, filtered 3010, and sent to an externalmusic synthesizer 3020. A filter 3010 can perform an operation on itsinput 3000 and pass the result on to its output 3010. Specific filterscan be written, for example, to add extra notes to create a MIDI echo,delay, or to prune pitch band to reduce bandwidth load.

FIG. 31 shows a media component that contains both MIDI and audio ports.A software-based music synthesizer reads MIDI packets from its inputport and outputs digital audio that represents the notes read on theinput port. MIDI ports are connected together by means of portsurrogates. Every MIDI component has member functions which createsurrogates for its input and output ports. Clients perform connectionoperations by requesting input and/or output port surrogates from eachcomponent and then using the member functions provided by the surrogateobjects to connect the actual input ports to actual output ports. EachMIDI port and port surrogate has a MIDI type associated with it. Whentwo ports are asked to connect, a type negotiation protocol insures thatthe ports are capable of supporting compatible data types. An exceptionis generated if the ports have no types in common.

The process of connecting MIDI components can be represented graphicallyon a display by extending a geometric figure such as a line segmentbetween the indicia representative of a MIDI input and output ports.

MIDI Packets

Devices that support MIDI communicate with each other by sending andreceiving MIDI messages. The MIDI standard defines two types ofmessages: channel messages and system messages. Channel messages arefurther divided into voice and mode messages. System messages arefurther divided into common, real-time, and exclusive messages as shownin FIG. 32. Channel Voice messages contain a channel number (0-15), towhich a MIDI device can listen. Channel Mode messages are sent on thebasic-channel to determine an instrument's response to Channel Voicemessages. System Common messages go to all receivers, and SystemReal-time messages carry synchronization information to clock-basedinstruments. System exclusive messages allow manufacturers to offer MIDIsupport beyond that specified by the standard. All messages start with astatus byte, except consecutive messages of the same type which mayoptionally drop the status byte (running status). All message typesexcept system exclusive, have zero, one, or two data bytes. Systemexclusive messages consist of any number of data bytes, terminated by anEOX byte. FIG. 33 exhibits formats of MIDI messages in accordance with apreferred embodiment.

MIDI Packet Encapsulates the Standard

A MIDI packet object encapsulates all the MIDI message types andstructures. In addition, all MIDI packet objects have a status byte anda time stamp, as shown in FIG. 34. Subclasses of MIDI packets reflectthe MIDI protocol by defining message types with convenient constructorsand access member functions.

MIDI Ports

MIDI ports are used to exchange MIDI packets between media components. AMIDI output port can write a MIDI packet, and a MIDI input port can reada MIDI packet. An output port can be connected to an input port with asurrogate port. Surrogate ports cannot read or write MIDI packets.Instead they are passive objects that can be passed to other addressspaces to support connections. Member functions are provided for writingone or many messages at a time to an output port. Similarly, whenreading from an input port, either the next message or a count of allbuffered messages may be requested. Read will block until the buffer isnon-empty, and a blocked Read call can be canceled by another task. Acopy of the packet written to an output port is read from the inputport.

Packets are read in order of arrival and are not sorted by time stamp.If time-ordered merging of two streams is required, a sorting mergeobject will be needed. To understand why an input port does not sort,recall first that time can flow both forwards and backwards. Consider aparticular playback scenario involving a sequence with events at 6, 8,and 10 seconds. If time starts at 5, goes until 11, turns around, andgoes back to 5, the order of events should be 6, 8, 10, 10, 8, 6 and not6, 6, 8, 8, 10, 10. It is precisely because time goes in both directionsthat sorting does not suffice for buffering packets.

Fan-in and fan-out of connections is supported. Fanning refers to thecapability of a MIDI output port to be connected to more than one MIDIinput port, and a MIDI input port can have more than one MIDI outputport connected to it. FIG. 35 illustrates an example of a fanningoperation in accordance with a preferred embodiment.

Implementing MIDI Ports

A preferred embodiment utilizes a list of connected ports for keepingtrack of MIDI port connections and a shared buffer for delivering andreceiving packets. Each port maintains a list of all the other ports towhich it is connected. This list is updated by Connect and Disconnectcalls and is used when ports are destroyed to notify all ports that areconnected to it that have gone away. This processing prevents an outputport from having an input port in its list of connections and trying towrite to it, when in fact the input port was destroyed. Thus, an outputport maintains a list of input ports that it uses to implement the Writecall, and it writes to each port in its list. And an input portmaintains a list of output ports that are connected to it and that mustbe notified when it goes away. A shared buffer supports aproducer-consumer protocol. When the buffer is empty, a task may blockuntil it is non empty. Another task that subsequently deposits into itsbuffer may notify the blocked task that it has written. Alternatively,yet another task may cancel the block, resulting in another kind ofnotification.

A MIDI output port's Write( ) member function is depicted in FIG. 36.Processing commences at terminal 3600 and immediately passes to functionblock 3610 to initialize a counter, and a limit value for a loop. Then,at decision block 3620, a test is performed to determine if the counterhas reached the limit value. If the counter has, then processing iscompleted at terminal 3670. If the counter has not, then at functionblock 3630, a copy of the packet is inserted into a port's buffer, and atest is performed at decision block 3640 to determine if the buffer hasa message. If the buffer is not empty, then an appropriate message istransmitted as shown in function block 3650. If not, then no message issent. In either case, the counter is incremented at function block 3660and control is returned via terminal 3670.

A MIDI input port's Read( ) member function is depicted in FIG. 37.Processing commences at decision block 3700 to determine if the bufferis empty. If so, then the task is blocked until the buffer containsinformation or a cancel occurs at function block 3710. A test isperformed at decision block 3730 to determine if the wait was cancelled.If so, then an exception occurs at terminal 3750. If not, or if thebuffer was not empty at decision block 3700, then a packet is copiedfrom the buffer to the caller at function block 3720 and processing iscompleted by returning at terminal 3740.

Abstract MultiMedia Components Media Component

A time-based media component, referred to as a media component baseclass is a central abstraction used for routing. A media component haszero or more input ports and zero or more output ports. In FIG. 38, forexample, the media component has a single input port and two outputports.

Media Sequence

Mediasequence is an abstract base class that represents media content,including audio sequences. Subclasses of mediasequence are used torepresent clips of audio, video, and MIDI. Media sequences arecharacterized by a duration and a list of types. The duration,represented by a floating point value, indicative of how long the datais. The data is also typed, to indicate what type of sound isrepresented by the data, for example video, audio, etc. It is possiblefor a subclass to support multiple types. For example, an audio subclasscan supply data in both a linear form and a compressed form. Because ofthis possibility, mediasequence has a list of types.

Player

FIG. 39 presents an example of a time-based media player (player) baseclass is a media component that can play time-based media sequences(referred to as a media sequence). A media sequence is an abstract baseclass that can be used to represent a clip of audio, video, animation,or MIDI data. Subclasses of the time-based media sequence are used toimplement audio, video, animation, and MIDI sequences. A player isanalogous to a tape recorder while a media sequence is analogous to acassette tape. A player has a Play( ) member function to play the mediasequence, a Record( ) member function to record into the sequence, and aStop( ) member function to stop playback or recording. It also has aSeek( ) member function to seek to a position in the sequence, as wellas member functions to allow the player to synchronize to other playersor to software clocks. Separating players from data also allows playersto be reused. After playing one cassette, a player can then beinstructed to play or record another.

Standard Audio Components Audio Player

FIG. 40 illustrates an audio player, which has an associated abstractbase class, for an audio component to play and record audio data. Audiodata is stored in objects called audio sequences, which are subclassesof media sequences. An audio player is analogous to a tape recorder andthe audio sequence is analogous to a cassette. An audio player is asubclass of a player. Like all players—sound, video, or MIDI—an audioplayer has a Play( ) member function, so that the sound can be heard,and a Record( ) member function, so the sequence can be recorded (ifwriting is allowed), and Stop( ) to stop recording or playing. Is has aSeek( ) member function to randomly access a point in the sound. It alsohas member functions to allow the player to be synchronized to anothermedia player or a software clock.

Speaker

FIG. 41 illustrates an example of a speaker component. A speakercomponent is an abstract base class for an audio output device. A systemmay have several sound output devices connected to it. A speakercomponent can be subclassed to represent various features of the outputdevices. For example, a subclass of a speaker component exists to allowplayback over the telephone handset or telephone line. A speakercomponent is a logical output device—many instances of speakercomponents can exist—all are mixed together and played out on thecomputer's physical speaker. Stereo output devices would be representedby two subclasses of the speaker component, one for the left channel andone for the right.

Microphone

FIG. 42 illustrates a microphone component in accordance with apreferred embodiment. A microphone component is an abstract base classrepresenting a sound input source, such as a microphone or a line input.A system may have several sound input devices connected to it. Amicrophone component can be subclassed to represent these devices. Forexample, a subclass of the microphone component is used to facilitaterecording from the telephone handset or telephone line. Like a speakercomponent, the microphone component is a logical device—multipleinstances of a microphone component can exist, all instances produceidentical audio which comes from the same physical microphone. Amicrophone component has one output. Because a microphone component is asource of data, you must start it explicitly by calling Start( ). It canbe stopped by calling Stop( ). Stereo input devices are represented bytwo subclasses of a microphone component, one for the left channel andone for the right.

Mixer

FIG. 43 illustrates an example of a mixer component. A mixer componentis an abstract base class that sums two or more audio inputs into asingle audio output. The number of inputs is determined by the input tothe constructor.

Splitter

FIG. 44 illustrates an example of a splitter component. A splittercomponent is an abstract base class that takes one input and splits itinto two or more outputs. The number of outputs is determined by what ispassed to the constructor.

Gain

FIG. 45 illustrates an example of a gain component. A gain component isan abstract base class that can increase or decrease the amplitude(size) of a signal. It is used for volume control, and has SetGain( )and GetGain( ) functions. Mathematically, a gain component multiplieseach input sound sample by the gain value and feeds the result to itsoutput. A gain value of 1.0 doesn't modify the signal at all, while again of 2.0 makes it twice as loud, 0.5 as a gain value changes thesignal to half as loud. Large signal levels are clipped.

Echo

FIG. 46 illustrates an echo component in accordance with a preferredembodiment. An echo component is an abstract base class that adds anecho to its input and produces the result on its output. An echocomponent has three parameters you can set, the delay length, feedback,and the mix ratio. Large delay lengths make the input sound like it's ina large canyon, while small delays can produce a flanging effect similarto the whooshing of a jet airplane. The feedback determines the numberof echoes heard. The mix ratio determines how much of the echoed signalis mixed back in.

Fuzz

FIG. 47 illustrates a fuzz component in accordance with a preferredembodiment. A fuzz component is an abstract base class that addsdistortion to a sound. This effect is most useful on guitar or musicalinstrument sounds.

Audio Type Converter

FIG. 48 illustrates an example of an audio type converter. An audio typeconverter component is an abstract base class that converts from oneaudio type to another.

Audio MultiConverter

FIG. 49 illustrates an audio multiconverter in accordance with apreferred embodiment. An audio multiconverter component converts frommultiple audio types to a plurality of other audio types based on aparticular selection.

Sound

FIG. 50 illustrates a sound component in accordance with a preferredembodiment. A sound component is a convenience object useful forrecording and playing sound. FIG. 51 illustrates the components imbeddedin a sound component. A sound component 5100 contains a microphonecomponent 5110, a gain component 5120 (to control input level), an audioplayer component 5130, another gain component 5140 (to control outputlevel) and a speaker component 5150. Given a sound file name, a soundcomponent will automatically create the correct audio sequence and thenecessary audio components to play and record the file.

Physical Speaker

FIG. 52 illustrates a physical speaker component in accordance with apreferred embodiment. A physical speaker component is a surrogate forthe computer's output amplifier and speaker. It is used to controlvolume for the entire computer. If the computer has other sound outputdevices, corresponding subclasses of a physical speaker would exist.Physical speaker components have SetVolume( ) and GetVolume( ) memberfunctions to set and get the overall volume level.

Physical Microphone

FIG. 53 illustrates a physical microphone component in accordance with apreferred embodiment. A physical microphone component represents theactual hardware for a microphone or line input. The overall input levelcoming into the computer can be set.

Standard Video Components Graphic Player

FIG. 54 illustrates a graphic player component in accordance with apreferred embodiment. A graphic player component is an abstract baseclass for a component that can play and record time graphics objectswhich are graphic objects that vary over time. A graphic sequence is asubclass of time graphic. A graphic sequence is an ordered sequence ofgraphic objects, here each graphic object has a duration. An graphicplayer is analogous to a video tape recorder and the graphic sequence isanalogous to a video cassette. A graphic player is a subclass of aplayer. Like all players—sound, video, or MIDI—a graphic player has aPlay( ) member function, so that the graphic objects can be seen, and aRecord( ) member function, so the sequence can be recorded (if writingis allowed), and Stop( ) to stop recording or playing. Is has a Seek( )member function to randomly access a point in the sequence. It also hasmember functions to allow the player to be synchronized to another mediaplayer or a software clock.

Graphic Viewer

FIG. 55 illustrates a graphic viewer component in accordance with apreferred embodiment. A graphic viewer component is used to view graphicobjects on a computer display. A graphic player must be connected to agraphic viewer in order to see the played graphic objects. Using ananalogy to consumer electronic video, a graphic player is like videotape recorder, while a graphic viewer is like a video display monitor.

Video Digitizer

FIG. 56 illustrates a video digitizer component in accordance with apreferred embodiment. A video digitizer component converts analog videodigitized by a hardware video digitizer connected to the computer intographic objects. A video digitizer can be connected to a graphic playerto record analog video entering the computer. Using an analogy to aconsumer electronic video, the video digitizer is analogous to a videocamera while the graphic player is analogous to a video tape recorder.

Standard MIDI Components MIDI Player

FIG. 57 illustrates a MIDI player component in accordance with apreferred embodiment. A MIDI player component is an abstract base classfor a component that can play and record MIDI Sequences. A MIDI sequenceis a collection of MIDI Tracks. A MIDI Track is an ordered sequence MIDIPackets. A MIDI player is a subclass of a player. Like allplayers—sound, video, or MIDI—a MIDI player has a Play( ) memberfunction, so that MIDI Packets can be sent to an external musicsynthesizer, and a Record( ) member function, so that MIDI messages froman external keyboard can be recorded (if writing is allowed), and Stop() to stop recording or playing. Is has a Seek( ) member function torandomly access a point in the sequence. It also has member functions toallow the player to be synchronized to another media player or asoftware clock.

MIDI Interface

FIG. 58 illustrates a MIDI interface component in accordance with apreferred embodiment. A MIDI interface component both sends MIDI packetsto an external music synthesizer and receives them. A MIDI player mustbe connected to a MIDI interface in order to play or record MIDImessages.

MIDI Filter

FIG. 59 illustrates a MIDI filter component in accordance with apreferred embodiment. A MIDI filter component is an abstract base classfor an object that has one MIDI input port and one MIDI output port.Subclasses supply a transformation algorithm that converts the inputdata to output data.

MIDI Mapper

FIG. 60 illustrates a MIDI mapper component in accordance with apreferred embodiment. The MIDI mapper component is a subclass of MIDIFilter that uses a dictionary to map input MIDI packets to output MIDIpackets. Each entry in the dictionary is a pair of MIDI packets, aninput packet (called the key) and an output packet (called the value).Input MIDI packets coming into the MIDI mapper are used as look up keysto the dictionary. The result of the lookup is the output MIDI packet,which is written out the output port.

MIDI Program Mapper

FIG. 61 illustrates a MIDI program mapper component in accordance with apreferred embodiment. A MIDI program mapper component is a subclass ofMIDI Mapper. It converts MIDI program change messages to other MIDIprogram change messages. It can be used to map any instrument to anyother.

MIDI Note Mapper

FIG. 62 illustrates a MIDI note mapper component in accordance with apreferred embodiment. A MIDI note mapper component is a subclass of MIDIMapper. It converts MIDI note on and note off messages, allowing notesto be transposed.

MIDI Channel Mapper

FIG. 63 illustrates a MIDI channel mapper component in accordance with apreferred embodiment. A MIDI channel mapper component is a subclass ofMIDI Mapper. It converts MIDI channel voice and mode messages and can beused for general purpose channel changing.

While the invention has been described in terms of a preferredembodiment in a specific system environment, those skilled in the artrecognize that the invention can be practiced, with modification, inother and different hardware and software environments within the spiritand scope of the appended claims.

1. A user-configurable multimedia information processing system for useon a computer having a display, the system comprising: a media sourcethat generates a multimedia information stream composed of a timesequence of multimedia information packets and that comprises amechanism that generates an icon on the display representing the mediasource, an output port having a write method that sequentially writeseach multimedia information packet to a first buffer and a connectmethod that connects the output port to an input port specified by auser; a media processing component that sequentially processesmultimedia information packets comprising a mechanism that generates anicon on the display representing the media processing component, aprocessing task that processes multimedia information packets, an inputport with a read method that reads multimedia information packets intothe processing task and an output port with a write method that receivesprocessed multimedia information packets from the processing task andsequentially writes the processed multimedia information packets to asecond buffer and a connect method that connects the output port to aninput port specified by a user; a media presentation component thatsequentially presents multimedia information packets to a user andcomprises a mechanism that generates an icon on the display representingthe media presentation component, an input port with a read method thatreads multimedia information packets from a buffer for presentation tothe user; and a mechanism that allows a user to manipulate icons on thedisplay to call the media source connect method and the processingcomponent connect method to selectively connect the media source, theprocessing component and the media presentation component to form amultimedia information processing system.
 2. The system of claim 1wherein each input port comprises a data type and each output portcomprises a data type and wherein each connect method comprises a typenegotiator that examines the data types of an input port and an outputport in order to determine whether that input port and that output portcan be connected.
 3. The system of claim 1 wherein the mediapresentation component is connected to the media source and the mediapresentation input port reads multimedia information packets from thefirst buffer for presentation to the user.
 4. The system of claim 1wherein the media source is connected to the media processing componentand that media presentation component is connected to the mediaprocessing component and the media presentation input port readsmultimedia information packets from the second buffer for presentationto the user.